IF: Create new efficient incremental Merkle tree #2361
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For 10 levels of recursion, we used ~1540 bytes of stack. This translates to 5KB of stack space (32 levels of recursion) for appending 4 billion digests (the max. possible number of blocks). Since the default Ubuntu stack for a process is 10MB, and 2MB for a thread, this means that, even for a thread, we'd use at most 5/2000, or 0.25% of the stack available for any append(). If the stack is already 99.75% used before `incremental_merkle_tree::append()` is called, we have a problem elsewhere.
…acy` and simplify `unittests/merkle_tree_tests.cpp`
…what `mask` means.
heifner
requested changes
Mar 30, 2024
| return hash_combine(start[0], start[1]); | ||
| else { | ||
| if (async && size >= 4096) { // below 4096, starting async threads is overkill | ||
| std::array<std::future<digest_type>, 4> fut; // size dictates the number of threads (must be power of two) |
There was a problem hiding this comment.
This is likely fine as it is only 4 threads. We do need to be mindful of thread usage as we don't want to overwhelm a system so that it slows down the main thread. Other thread usage is setup at startup. Would be nice to be able to provide an io_context for these tasks so they could run on the chain thread pool; however likely not worth the effort.
There was a problem hiding this comment.
I updated it so that it uses only 2 threads up to 4096 digests, and 4 for 4096 and up.
libraries/chain/include/eosio/chain/incremental_merkle_legacy.hpp
Outdated
Show resolved
Hide resolved
|
Note:start |
heifner
approved these changes
Apr 1, 2024
linh2931
approved these changes
Apr 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #2333
This PR provides a new Merkle tree implementation which does not unnecessarily store extra digests and does not do extra hashes.
Previous versions are renamed:
incremental_merkle_tree_legacy,calculate_merkle_legacy.New versions are named:
incremental_merkle_tree,calculate_merkle.Note: Even though the files have been renamed
incremental_merkle_legacy.hppandmerkle_legacy.hpp, the implementation has not been modified (besides remoning thecanonicaltemplate parameter).New
incremental_merkle_treecalculate_merkle)append().incremental_merkle_tree_legacy.uint64_tand avector<digest_type>whose size() isstd::popcount(num_digest_appended).New
calculate_merkleincremental_merkle_tree).calculate_merkle_legacywithout multithreadingsize() > 4096, in which case it is 2x to almost 4x faster (2x for sequences of at least 10k digests, 4x for very large sequences).deque<digest_type>.New performance tests
This PR adds two performance tests comparing the
legacyandnewmerkle tree implementations./unittests/unit_test "--run_test=merkle_tree_tests/perf_test_one_large"./unittests/unit_test "--run_test=merkle_tree_tests/perf_test_many_small"New consistency test
This PR adds a test checking consistency between
incremental_merkle_treeandcalculate_merklewhen adding from 1 to 1000 digests:./unittests/unit_test "--run_test=merkle_tree_tests/consistency_over_large_range"